A classical study by Blakemore & Cooper 1970 reported that edge detectors are far from being totally innate. If kittens were raised in an environment with only vertical edges, only neurons that respond to vertical or nearly vertical edges were found in the primary visual cortex. Recently, experimental studies have shown how the cortex develops through input-driven self-organization, in a dynamic equilibrium with internal and external inputs. Such dynamic development and adaption occurs from the prenatal stage and continues throughout infancy, childhood, and adulthood. Recently, it has been reported that the spatio-temporal receptive fields and the response of retinal ganglion cells change after a few seconds in a new environment. The changes are adaptive in that the new receptive field improves predictive coding under the new image statistics.
Orientation selective cells in the cortical area V1 are well known, but the underlying learning mechanisms that govern their emergence (i.e., development) are still elusive. Furthermore, complex cells and other cells in V1 that do not exhibit a clear orientation are still poorly understood in terms of their functions and their underlying learning mechanisms. Understanding cortical filter development algorithms is important to address these open problems.
A network is developmental if it can deal with autonomous epigenetic development—namely, it has fully automatic internal self-organization that enables it to autonomously learn a plurality of sensorimotor skills or a plurality of tasks through autonomous interactions with its environment (which may include humans). The work for filter development reported here is fundamentally different from hand-designed filters such as Gabor filters, because the lobe component analysis (LCA) filters presented here are automatically generated (i.e., developed) form signals. Unlike hand-designed filters (e.g., Gabor filters) which wastefully tile the entire input space which is only sparsely sensed, LCA only develops filters to optimally cover all the subparts of input space that have been sensed. Because of this “spectrum finding” property and its use of biological mechanisms found in various cortices, LCA can be potentially used to develop filters for any cortical area e.g., any sensory modalities (vision or audition) for their early, intermediate, or later processing.
In order to understand what the biological cortex detects, how it represents internally and how it develops feature detectors, this disclosure rigorously proposes a classification of learning algorithms, and introduces the concept of in-place learning. Then, it introduces a biologically inspired new concept, called lobe components.
Consider a neuron which takes n inputs, x=(x1, x21, . . . xn). The synaptic weight for xi is wi, i=1, 2, . . . , n or write w=(w1, w2, . . . , wn). The response l of a neuron has been modeled by l=g (w·x), where g is a monotone, nonlinear, sigmoid function. The learning (adaptation) of the neuron is characterized by the modification of g and w using input vectors xt, t=1, 2, . . . .
To facilitate understanding of the nature of learning algorithms, we define five types of learning algorithms:                Type-1 batch: A batch learning algorithm L1 computes g and w using a batch of vector inputs B={x1, x2, . . . , xb}, where b is the batch size.(g,w)=L1(B),  (1)        where the argument B on the right is the input to the learner, L1 and the right side is its output. The learning algorithm needs to store an entire batch to input vectors B before learning can take place. Since L1 requires the additional storage of B, L1 must be realized by a separate network L1 and thus, the learning of the (learning) network L1 is an open problem.        Type-2 block-incremental: A type-2 learning algorithm, L2, breaks a series of input vectors into blocks of certain size b (b>1) and computes updates incrementally between blocks. Within each block, the processing by L2 is in a batch fashion.        Type-3 incremental: Each input vector must be used immediately for updating the learner's memory (which must not store all the input vectors) and then the input must be discarded before receiving the next input. Type-3 is the extreme case of Type-2 in the sense that block size b=1. A type-2 algorithm, such as Infomax, becomes a Type-3 algorithm by setting b=1, but the performance will further suffer.        Type-4 covariance-free incremental: A Type-4 learning algorithm L4 is a Type-3 algorithm, but furthermore, it is not allowed to compute the 2nd or higher order statistics of the input x. In other words, the learner's memory M(t) cannot contain the second order (e.g., correlation or covariance) or higher order statistics of x. The CCI PCA algorithm is a covariance-free incremental learning algorithm for computing principal components as the weight vectors of neurons. Further information regarding this algorithm may be found in a paper by Weng et al entitled “Candid Covariance-free Incremental Principle Component Analysis” IEEE Trans. Pattern Analysis and Machine Intelligence, 25(8):1034-1040 (2003).        Type-5 in-place neuron learning: A Type-5 learning algorithm L5 is a Type-4 algorithm, but further the learner L5 must be implemented by the signal processing neuron N itself. A term “local learning” used by some researchers does not imply in-place. For example, for a neuron N, its signal processor model has two parts, the synaptic weight w(t) and the sigmoidal function g(t), both updated up to t. A type-5 learning algorithm L5 must update them using the previously updated weight w(t−1) and the sigmoidal function g(t−1), using the current input xt while keeping its maturity indicated by t:(w(t),g(t),t)=L5(w(t−1),g(t−1),t−1,xt).  (2)        After the adaptation, the computation of the response is realized by the neuron N:yt=g(t)(w(t)·xt).  (3)        An in-place learning algorithm must realize L5 and the computation above by the same neuron N, for t=1, 2, . . . .        
Principal components computed from natural images can be used as the weight vectors of neurons. Some of the principal components have a clear orientation, but their orientations are crude (e.g., exhibiting few directions) and their support is not localized (i.e., most synapses have significant magnitudes).
Higher order statistics (e.g., higher-order moments, kurtosis, etc.) have been used in the Independent Component Analysis (ICA). The ICA algorithms search, from higher order statistics of inputs, a set of filters whose responses are statistically as independent as possible. However, the published ICA algorithms are Type-1 and Type-2 algorithms. By these algorithms, every learning neuron requires a dedicated, separate neural network to handle its learning. However, the existence and learning of this separate learning network has largely been left unaddressed. This overlooked obvious problem needs to be resolved not only for any neural learning algorithm to be biologically plausible, but more importantly, for understanding the tightly associated signal processing network.
The concept of in-place learning is based on the following hypothesis. Each neuron with n synaptic weights does not have extra space in itself to store correlation values and other higher-order moments of its input. For example, the covariance matrix of input x requires (n+1)n/2 additional storage units. A typical neuron in the brain has about n=1000 synapses. The correlations of these n input fibres requires (n+1)n/2=500,500 storage units. A Purkinje cell typically has about n=150,000 synapses. The correlation of its input vector requires as many as (n+1)n/2≈1.125×1010 storage units. It is unlikely that a neuron has so many storage units within its cell body. For example, there is no evidence that the genes, residing in the cell nucleus, record and recall all correlations of cells input fibres.
Surprisingly, by forcing the learning algorithm to be in place, we arrive at a deeper understanding of not only how they learn, but also their functions.